Cell segmentation image processing methods

ABSTRACT

Methods for the image processing od cells are disclosed for the purpose of segmenting cell colonies for further processing to determine confluence, counting and morphology.

FIELD OF THE INVENTION

The present invention relates to imaging systems and in particular toimaging systems for cell cultures.

BACKGROUND

Cell culture incubators are used to grow and maintain cells from cellculture, which is the process by which cells are grown under controlledconditions. Cell culture vessels containing cells are stored within theincubator, which maintains conditions such as temperature and gasmixture that are suitable for cell growth. Cell imagers take images ofindividual or groups of cells for cell analysis.

Cell culture is a useful technique in both research and clinicalcontexts. However, maintenance of cell cultures, for example, long termcultures, tissue preparations, in vitro fertilization preparations,etc., in presently available cell incubators is a laborious processrequiring highly trained personnel and stringent aseptic conditions.

While scientists use microscopes to observe cells during culturing andmay also attach a camera to the microscope to image cells in a cellculture, such imaging systems have many disadvantages.

SUMMARY

The object of the present invention is to provide an improved imagingsystem and method for cells in a cell culture. An imaging system andmethod of this type is described in U.S. Application serial number15/563,375 filed on Mar. 31, 2016 and the disclosure of which in itsentirety is hereby incorporated by reference.

In some embodiments, the imaging system and method described herein canbe used as a stand-alone imaging system or it can be integrated in acell incubator using a transport described in the aforementionedapplication incorporated by reference. In some embodiments, the imagingsystem and method is integrated in a cell incubator and includes atransport.

In some embodiments the system and method acquire data and images at thetimes a cell culturist typically examines cells. The method and systemprovide objective data, images, guidance and documentation that improvescell culture process monitoring and decision-making.

The system and method in some embodiments enable sharing of bestpractices across labs, assured repeatability of process across operatorsand sites, traceability of process and quality control. In someembodiments the method and system provide quantitative measures of celldoubling rates, documentation and recording of cell morphology,distribution and heterogeneity.

In some embodiments, the method and system provide assurance that celllines are treated consistently and that conditions and outcomes aretracked. In some embodiments the method and system learn throughobservation and records how different cells grow under controlledconditions in an onboard database. Leveraging this database ofobservations, researchers are able to profile cell growth, testpredictions and hypotheses concerning cell conditions, media and otherfactors affecting cell metabolism, and determine whether cells arebehaving consistently and/or changing.

In some embodiments the method and system enable routine and accurateconfluence measurements and imaging and enables biologists to quantifyresponses to stimulus or intervention, such as the administration of atherapeutic to a cell line.

The method and system capture the entire well area with higher coveragethan conventional images and enables the highest level of statisticalrigor for quantifying cell status and distribution.

In some embodiments, the method and system provide image processing andalgorithms that will deliver an integration of individual and groupmorphologies with process-flow information and biological outcomes. Fullwell imaging allows the analysis and modeling of features of groups ofcells - conducive to modeling organizational structures in biologicaldevelopment. These capabilities can be used for prediction of theorganizational tendency of culture in advance of functional testing.

In some embodiments, algorithms are used to separate organizationalpatterns between samples using frequency of local slope fieldinversions. Using some algorithms, the method and system canstatistically distinguish key observed differences between iP-MSCsgenerated from different TCP conditions. Biologically, this work couldvalidate serum-free differentiation methods for iPSC MSCdifferentiation. Computationally, the method and system can informimage-processing of MSCs in ways that less neatly “clustered” image setsare not as qualified to do.

Even if all iP-MSC conditions have a sub-population of cells that meetsISCT 7-marker criteria, the “true MSC” sub-populations may occupy adifferent proportion under different conditions or fate differencescould be implied by tissue “meso-structures”. By starting with a richpallet of MSC outcomes, and grounding them in comparative biologicaltruth, the method and system can refine characterization perspectivesaround this complex cell type and improve MSC bioprocess.

In certain embodiments, an imager includes one or more lenses, fibers,cameras (e.g., a charge-coupled device camera), apertures, mirrors,light sources (e.g., a laser or lamp), or other optical elements. Animager may be a microscope. In some embodiments, the imager is abright-field microscope. In other embodiments, the imager is aholographic imager or microscope. In other embodiments the imager is aphase-contrast microscope. In other embodiments, the imager is afluorescence imager or microscope.

As used herein, the fluorescence imager is an imager which is able todetect light emitted from fluorescent markers present either within oron the surface of cells or other biological entities, said markersemitting light in a specific wavelength when absorbing a light ofdifferent specific excitation wavelength.

As used herein, a “bright-field microscope” is an imager thatilluminates a sample and produces an image based on the light absorbedby the sample. Any appropriate bright-field microscope may be used incombination with an incubator provided herein.

As used herein, a “phase-contrast microscope” is an imager that convertsphase shifts in light passing through a transparent specimen tobrightness changes in the image. Phase shifts themselves are invisiblebut become visible when shown as brightness variations. Any appropriatephase-contrast microscope may be used in combination with an incubatorprovided herein.

As used herein, a “holographic imager” is an imager that providesinformation about an object (e.g., sample) by measuring both intensityand phase information of electromagnetic radiation (e.g., a wave front).For example, a holographic microscope measures both the lighttransmitted after passing through a sample as well as the interferencepattern (e.g., phase information) obtained by combining the beam oflight transmitted through the sample with a reference beam.

A holographic imager may also be a device that records, via one or moreradiation detectors, the pattern of electromagnetic radiation, from asubstantially coherent source, diffracted or scattered directly by theobjects to be imaged, without interfering with a separate reference beamand with or without any refractive or reflective optical elementsbetween the substantially coherent source and the radiation detector(s).

Holographic Microscopy

In some embodiments, holographic microscopy is used to obtain images(e.g., a collection of three-dimensional microscopic images) of cellsfor analysis (e.g., cell counting) during culture (e.g., long-termculture) in an incubator (e.g., within an internal chamber of anincubator as described herein). In some embodiments, a holographic imageis created by using a light field, from a light source scattered off objects, which is recorded and reconstructed. In some embodiments, thereconstructed image can be analyzed for a myriad of features relating tothe objects. In some embodiments, methods provided herein involveholographic interferometric metrology techniques that allow fornon-invasive, marker-free, quick, full-field analysis of cells,generating a high resolution, multi-focus, three-dimensionalrepresentation of living cells in real time.

In some embodiments, holography involves shining a coherent light beamthrough a beam splitter, which divides the light into two equal beams: areference beam and an illumination beam. In some embodiments, thereference beam, often with the use of a mirror, is redirected to shinedirectly into the recording device without contacting the object to beviewed. In some embodiments, the illumination beam is also directed,using mirrors, so that it illuminates the object, causing the light toscatter. In some embodiments, some of the scattered light is thenreflected onto the recording device. In some embodiments, a laser isgenerally used as the light source because it has a fixed wavelength andcan be precisely controlled. In some embodiments, to obtain clearimages, holographic microscopy is often conducted in the dark or in lowlight of a different wavelength than that of the laser in order toprevent any interference. In some embodiments, the two beams reach therecording device, where they intersect and interfere with one another.In some embodiments, the interference pattern is recorded and is laterused to reconstruct the original image. In some embodiments, theresulting image can be examined from a range of different angles, as ifit was still present, allowing for greater analysis and informationattainment.

In some embodiments, digital holographic microscopy is used inincubators described herein. In some embodiments, digital holographicmicroscopy light wave front information from an object is digitallyrecorded as a hologram, which is then analyzed by a computer with anumerical reconstruction algorithm. In some embodiments, the computeralgorithm replaces an image forming lens of traditional microscopy. Theobject wave front is created by the object’s illumination by the objectbeam. In some embodiments, a microscope objective collects the objectwave front, where the two wave fronts interfere with one another,creating the hologram. Then, the digitally recorded hologram istransferred via an interface (e.g., IEEE1394, Ethernet, serial) to aPC-based numerical reconstruction algorithm, which results in a viewableimage of the object in any plane.

In some embodiments, in order to procure digital holographic microscopicimages, specific materials are utilized. In some embodiments, anillumination source, generally a laser, is used as described herein. Insome embodiments, a Michelson interferometer is used for reflectiveobjects. In some embodiments, a Mach-Zehnder interferometer fortransmissive objects is used. In some embodiments, interferometers caninclude different apertures, attenuators, and polarization optics inorder to control the reference and object intensity ratio. In someembodiments, an image is then captured by a digital camera, whichdigitizes the holographic interference pattern. In some embodiments,pixel size is an important parameter to manage because pixel sizeinfluences image resolution. In some embodiments, an interferencepattern is digitized by a camera and then sent to a computer as atwo-dimensional array of integers with 8-bit or higher grayscaleresolution. In some embodiments, a computer’s reconstruction algorithmthen computes the holographic images, in addition to pre- andpost-processing of the images.

Phase Shift Image

In some embodiments, in addition to the bright field image generated, aphase shift image results. Phase shift images, which are topographicalimages of an object, include information about optical distances. Insome embodiments, the phase shift image provides information abouttransparent objects, such as living biological cells, without distortingthe bright field image. In some embodiments, digital holographicmicroscopy allows for both bright field and phase contrast images to begenerated without distortion. Also, both visualization andquantification of transparent objects without labeling is possible withdigital holographic microscopy. In some embodiments, the phase shiftimages from digital holographic microscopy can be segmented and analyzedby image analysis software using mathematical morphology, whereastraditional phase contrast or bright field images of living unstainedbiological cells often cannot be effectively analyzed by image analysissoftware.

In some embodiments, a hologram includes all of the informationpertinent to calculating a complete image stack. In some embodiments,since the object wave front is recorded from a variety of angles, theoptical characteristics of the object can be characterized, andtomography images of the object can be rendered. From the complete imagestack, a passive autofocus method can be used to select the focal plane,allowing for the rapid scanning and imaging of surfaces without anyvertical mechanical movement. Furthermore, a completely focused image ofthe object can be created by stitching the sub-images together fromdifferent focal planes. In some embodiments, a digital reconstructionalgorithm corrects any optical aberrations that may appear intraditional microscopy due to image-forming lenses. In some embodiments,digital holographic microscopy advantageously does not require a complexset of lenses; but rather, only inexpensive optics, and semiconductorcomponents are used in order to obtain a well-focused image, making itrelatively lower cost than traditional microscopy tools. Applications

In some embodiments, holographic microscopy can be used to analyzemultiple parameters simultaneously in cells, particularly living cells.In some embodiments, holographic microscopy can be used to analyzeliving cells, (e.g., responses to stimulated morphological changesassociated with drug, electrical, or thermal stimulation), to sortcells, and to monitor cell health. In some embodiments, digitalholographic microscopy counts cells and measures cell viability directlyfrom cell culture plates without cell labeling. In other embodiments,the imager can be used to examine apoptosis in different cell types, asthe refractive index changes associated with the apoptotic process canbe quantified via digital holographic microscopy. In some embodiments,digital holographic microscopy is used in research regarding the cellcycle and phase changes. In some embodiments, dry cell mass (which cancorrelate with the phase shift induced by cells), in addition to othernon-limiting measured parameters (e.g., cell volume, and the refractiveindex), can be used to provide more information about the cell cycle atkey points.

In some embodiments, the method is also used to examine the morphologyof different cells without labeling or staining. In some embodiments,digital holographic microscopy can be used to examine the celldifferentiation process; providing information to distinguish betweenvarious types of stem cells due to their differing morphologicalcharacteristics. In some embodiments, because digital holographicmicroscopy does not require labeling, different processes in real timecan be examined (e.g., changes in nerve cells due to cellularimbalances). In some embodiments, cell volume and concentration may bequantified, for example, through the use of digital holographicmicroscopy’s absorption and phase shift images. In some embodiments,phase shift images may be used to provide an unstained cell count. Insome embodiments, cells in suspension may be counted, monitored, andanalyzed using holographic microscopy.

In some embodiments, the time interval between image acquisitions isinfluenced by the performance of the image recording sensor. In someembodiments, digital holographic microscopy is used in time-lapseanalyses of living cells. For example, the analysis of shape variationsbetween cells in suspension can be monitored using digital holographicimages to compensate for defocus effects resulting from movement insuspension. In some embodiments, obtaining images directly before andafter contact with a surface allows for a clear visual of cell shape. Insome embodiments, a cell’s thickness before and after an event can bedetermined through several calculations involving the phase contrastimages and the cell’s integral refractive index. Phase contrast relieson different parts of the image having different refractive index,causing the light to traverse different areas of the sample withdifferent delays. In some embodiments, such as phase contrastmicroscopy, the out of phase component of the light effectively darkensand brightens particular areas and increases the contrast of the cellwith respect to the background. In some embodiments, cell division andmigration are examined through time-lapse images from digitalholographic microscopy. In some embodiments, cell death or apoptosis maybe examined through still or time-lapse images from digital holographicmicroscopy.

In some embodiments, digital holographic microscopy can be used fortomography, including but not limited to, the study of subcellularmotion, including in living tissues, without labeling.

In some embodiments, digital holographic microscopy does not involvelabeling and allows researchers to attain rapid phase shift images,allowing researchers to study the minute and transient properties ofcells, especially with respect to cell cycle changes and the effects ofpharmacological agents.

These and other features and advantages, which characterize the presentnon-limiting embodiments, will be apparent from a reading of thefollowing detailed description and a review of the associated drawings.It is to be understood that both the foregoing general description andthe following detailed description are explanatory only and are notrestrictive of the non-limiting embodiments as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view of the imaging system according to theinvention;

FIG. 2 is the imaging system of FIG. 1 with walls removed to reveal theinternal structure;

FIG. 3 is a top view of the imaging system of FIG. 1 with the wallsremoved;

FIG. 4 is a right side view of the imaging system of FIG. 1 ;

FIG. 5 is a left side view of the imaging system of FIG. 1 ;

FIG. 6 is a block diagram of the circuitry of the imaging system of FIG.1 ;

FIG. 7 is a not to scale diagram of the issues focusing on a plate withwells when it is in or out of calibration;

FIG. 8 is a not to scale diagram of a pre-scan focus method according tothe present invention when the plate is in and out of calibration;

FIGS. 9 a-9 d show the steps of one method of image processing accordingto the present invention;

FIGS. 10 a-10 c show different scenarios of the method of FIGS. 9 a-9 d;

FIG. 11 shows another step of the method of FIGS. 9 a-9 d ; and

FIG. 12 shows another method of image processing according to thepresent invention.

DETAILED DESCRIPTION

Referring now to FIG. 1 , a cell imaging system 10 is shown. Preferably,the system 10 is fully encased with walls 11 a-11 f so that the interiorof the imager can be set at 98.6° F. with a CO₂ content of 5%, so thatthe cells can remain in the imager without damage. The temperature andthe CO₂ content of the air in the system 10 is maintained by a gas feedport 14 (shown in FIG. 2 ) in the rear wall 11 e. Alternatively, aheating unit can be installed in the system 10 to maintain the propertemperature.

At the front wall 11 c of the system 10, is a door 12 that is hinged tothe wall 11 c and which opens a hole H through which the slidingplatform 13 exits to receive a plate and closes hole H when the platform13 is retracted into the system 10.

The system 10 can also be connected to a computer or tablet for datainput and output and for the control of the system. The connection is byway of an ethernet connector 15 in the rear wall 11 e of the system asshown in FIG. 2 .

FIG. 2 shows the system with walls 11 b and 11 c removed to show theinternal structure. The extent of the platform 13 is shown as well asthe circuit board 15 that contains much of the circuitry for the system,as will be explained in more detail hereinafter.

FIG. 3 shows a top view of the imaging system where plate P having sixwells is loaded for insertion into the system on platform 13. Motor 31draws the platform 13 and the loaded plate P into the system 10. Themotor 31 moves the platform 13 in both the X-direction into and out ofthe system and in the Y-direction by means of a mechanical transmission36. The movement of the platform is to cause each of the wells to beplaced under one of the LED light clusters 32 a, 32 b, and 32 c whichare aligned with microscope optics 33 a, 33 b and 33 c respectivelywhich are preferably 4X, 10X and 20X phase-contrast and brightfieldoptics which are shown in FIG. 4 .

As used herein, an “imager” refers to an imaging device for measuringlight (e.g., transmitted or scattered light), color, morphology, orother detectable parameters such as a number of elements or acombination thereof. An imager may also be referred to as an imagingdevice. In certain embodiments, an imager includes one or more lenses,fibers, cameras (e.g., a charge-coupled device or CMOS camera),apertures, mirrors, light sources (e.g., a laser or lamp), or otheroptical elements. An imager may be a microscope. In some embodiments,the imager is a bright-field microscope. In other embodiments, theimager is a holographic imager or microscope. In other embodiments, theimager is a fluorescence microscope.

As used herein, a “fluorescence microscope” refers to an imaging devicewhich is able to detect light emitted from fluorescent markers presenteither within and/or on the surface of cells or other biologicalentities, said markers emitting light at a specific wavelength inresponse to the absorption a light of a different wavelength.

As used herein, a “bright-field microscope” is an imager thatilluminates a sample and produces an image based on the light absorbedby the sample. Any appropriate bright-field microscope may be used incombination with an incubator provided herein.

As used herein, a “holographic imager” is an imager that providesinformation about an object (e.g., sample) by measuring both intensityand phase information of electromagnetic radiation (e.g., a wave front).For example, a holographic microscope measures both the lighttransmitted after passing through a sample as well as the interferencepattern (e.g., phase information) obtained by combining the beam oflight transmitted through the sample with a reference beam.

A holographic imager may also be a device that records, via one or moreradiation detectors, the pattern of electromagnetic radiation, from asubstantially coherent source, diffracted or scattered directly by theobjects to be imaged, without interfering with a separate reference beamand with or without any refractive or reflective optical elementsbetween the substantially coherent source and the radiation detector(s).

In some embodiments, an incubator cabinet includes a single imager. Insome embodiments, an incubator cabinet includes two imagers. In someembodiments, the two imagers are the same type of imager (e.g., twoholographic imagers or two bright-field microscopes). In someembodiments, the first imager is a bright-field microscope and thesecond imager is a holographic imager. In some embodiments, an incubatorcabinet comprises more than 2 imagers. In some embodiments, cell cultureincubators comprise three imagers. In some embodiments, cell cultureincubators having 3 imagers comprise a holographic microscope, abright-field microscope, and a fluorescence microscope.

As used herein, an “imaging location” is the location where an imagerimages one or more cells. For example, an imaging location may bedisposed above a light source and/or in vertical alignment with one ormore optical elements (e.g., lens, apertures, mirrors, objectives, andlight collectors).

Referring to FIGS. 4-5 , Under the control of the circuitry on board 15,each well is aligned with a desired one of the three optical units 33a-33 c and the corresponding LED is turned on for brightfieldillumination. The image seen by the optical unit is recorded by therespective video camera 35 a, 35 b, and 35 c corresponding to theoptical unit. The imaging and the storing of the images are all underthe control of the circuitry on board 15. After the imaging iscompleted, the platform with the loaded plate is ejected from the systemand the plate can be removed and placed in an incubator. Focusing of themicroscope optics is along the z axis and images taken at differentdistances along the z axis is called the z-stack.

FIG. 6 is a block diagram of the circuitry for controlling the system10. The system is run by processor 24 which is a microcontroller ormicroprocessor which has associated RAM 25 and ROM 26 for storage offirmware and data. The processor controls LED driver 23 which turns theLEDs on and off as required. The motor controller 21 moves the motor 15to position the wells in an imaging position as desired by the user. Ina preferred embodiment, the system can effect a quick scan of the platein less than 1 minute and a full scan in less than 4 minutes.

The circuitry also includes a temperature controller 28 for maintainingthe temperature at 98.6° F. The processor 24 is connected to an I/O 27that permits the system to be controlled by an external computer such asa laptop or desktop computer or a tablet such as an iPad or Androidtablet. The connection to an external computer allows the display of thedevice to act as a user interface and for image processing to take placeusing a more powerful processor and for image storage to be done on adrive having more capacity. Alternatively, the system can include adisplay 29 such as a tablet mounted on one face of the system and animage processor 22 and the RAM 25 can be increased to permit the systemto operate as a self-contained unit.

The image processing either on board or external, has algorithms forartificial intelligence and intelligent image analysis. The imageprocessing permits trend analysis and forecasting, documentation andreporting, live/dead cell counts, confluence percentage and growthrates, cell distribution and morphology changes, and the percentage ofdifferentiation.

When a new cell culture plate is imaged for the first time by themicroscope optics, a single z-stack, over a large focal range, of phasecontrast images is acquired from the center of each well using the 4xcamera. The z-height of the best focused image is determined using thefocusing method, described below. The best focus z-height for each wellin that specific cell culture plate is stored in the plate database inRAM 25 or in a remote computer. When a future image scan of that plateis done using either the 4x or 10x camera, in either brightfield orphase contrast imaging mode, the z-stack of images collected for eachwell are centered at the best focus z-height stored in the platedatabase. When a future image scan of that plate is done using the 20xcamera, a pre-scan of the center of each well using the 10x camera isperformed and the best focus z-height is stored in the plate database todefine the center of the z-stack for the 20x camera image acquisition.

Each whole well image is the result of the stitching together of anumber of tiles. The number of tiles needed depend on the size of thewell and the magnification of the camera objective. A single well in a6-well plate is the stitched result of 35 tiles from the 4x camera, 234tiles from the 10x camera, or 875 tiles from the 20x camera.

The higher magnification objective cameras have smaller optical depth,that is, the z-height range in which an object is in focus. To achievegood focus at higher magnification, a smaller z-offset needs to be used.As the magnification increases, the number of z-stack images needs toincrease or the working focal range needs to decrease. If the number ofz-stack images increase, more resources are required to acquire theimage, time, memory, processing power. If the focal range decreases, thelikelihood that the cell images will be out of focus is greater, due toinstrument calibration accuracy, cell culture plate variation, wellcoatings, etc.

In one implementation, the starting z-height value is determined by adatabase value assigned stored remotely or in local RAM. The z-height isa function of the cell culture plate type and manufacturer and is thesame for all instruments and all wells. Any variation in theinstruments, well plates, or coatings needs to be accommodated by alarge number of z-stacks to ensure that the cells are in the range offocus adjustment. In practice this results in large imaging times and isintolerance to variation, especially for higher magnification objectivecameras with smaller depth of field. For example, the 4x objectivecamera takes 5 z-stack images with a z-offset of 50 µm for a focal rangeof 5*50=250 µm. The 10x objective camera takes 11 z-stack images with az-offset of 20 µm for a focal range of 11 *20=220 µm. The 20x objectivecamera takes 11 z-stack images with a z-offset of 10 µm for a focalrange of 11*10=110 µm.

The processor 24 creates a new plate entry for each plate it scans. Theuser defines the plate type and manufacturer, the cell line, the wellcontents, and any additional experiment condition information. The userassigns a plate name and may choose to attach a barcode to the plate foreasier future handling. When that plate is first scanned, a pre-scan isperformed. For the pre-scan, the image processor 22 takes a z-stack ofimages of a single tile in the center of each well. The pre-scan usesthe phase contrast imaging mode to find the best focus image z-height.The pre-scan takes a large z-stack range so it will find the focalheight over a wider range of instrument, plate, and coating variation.The best focus z-height for each well is stored in the plate databasesuch that future scans of that well will use that value as the centervalue for the z-height.

Although the pre-scan method was described using the center of a well asthe portion where the optimal z-height is measured, it is understoodthat the method can be performed using other portions of the wells andthat the portion measured can be different or the same for each well ona plate.

In one embodiment, the 4x pre-scan takes 11 z-height images with az-offset of 50 µm for a focus range of 11*50=550 µm. For a 6-well plate,the 4x pre-scan takes 11 images per well, 6*11=66 images per plate. The4x pre-scan best focus z-heights are used for the 4x and 10x scans. Theadditional imaging is not significant compared to the 35*5*6=1050 imagesfor the 4x scan, and 234*11*6=15444 images for the 10x scan. For a 20xscan, the system performs a 10x pre-scan in addition to the 4x pre-scanto define the best focus z-height values to use as the 20x centerz-height value for the z-stacks. It is advantageous to limit the numberof pre-scan z-height measurements to avoid imaging the bottom plasticsurface of the well since it may have debris that could confuse thealgorithms.

As illustrated in FIGS. 7 and 8 , the pre-scan focus method relies onz-height information in the plate database to define the z-height valuesto image. Any variation in the instrument, well plate, or customerapplied coatings eats away at the z-stack range from which the focusedimage is derived, as shown in FIG. 7 . There is the possibility that thebest focus height will be outside of the z-stack range. The pre-scanmethod enables the z-stack range to be adjustable for each well, sodrooping of the plate holder, or variation of the plate, can beaccommodated within a wider range as shown in FIG. 8 .

A big advantage of this pre-scan focus method is that it can focus onwell bottoms without cells. For user projects like gene editing in whicha small number of cells are seeded, this is huge. In the pre-scan focusmethod, a phase contrast pre-scan enables the z-height range to be setcorrectly for a brightfield image.

Practical implementation of 10x and 20x cameras is difficult due to thesmall depth of field and the subsequent limited range of focus for areasonably sized z-stack. This pre-scan focus method enables the z-stackto be optimally centered around on the experimentally determinedz-height, providing a better chance of the focal plane being in range.

Since the z-stacks are centered around the experimentally determinedbest focus height, the size of the z-stack can be reduced. The reductionin the total number of images reduces the scan time, storage, andprocessing resources of the system.

In some embodiments, the pre-scan is most effective when performed in aparticular imaging mode, such as phase contrast. In such a circumstance,the optimal z-height determined using the pre-scan in that imaging modecan be applied to other imaging modes, such as brightfield,fluorescence, or luminescence.

In another embodiment, a method for segmentation of images of cellcolonies in wells is described. A demonstration of the method is shownin FIGS. 9 a-d . Three additional results from other raw images areshown in FIGS. 10 a-c that give an idea of the type of variation thealgorithm can now handle. The methods segment stem, cancer, and othercell colony types. The method manifests the following benefits: it isfaster to calculate than previous methods s based on spatial frequencysuch as Canny, Sobel, and localized Variance and entropy based methods;a single set of parameters serves well to find both cancer and stem cellcolonies; and the algorithm performs with different levels of confluenceand they do not mitigate the ability of the method to properly performsegmentation.

FIG. 9 a shows a raw image of low-confluence cancer cell colonies, FIG.9 b shows a remap image of FIG. 9 a in accordance with the algorithm,(a), FIG. 9 c shows a remap image of FIG. 9 b in accordance with thealgorithm, and FIG. 9 d shows the resulting contours in accordance withthe algorithm.

FIG. 10 shows example contours obtained from a method using thealgorithm for various scenarios. FIG. 10 a is the scenario of highconfluence cancer cells, FIG. 10 b is the scenario for low confluencestem cells, and FIG. 10 c is the scenario for medium confluence stemcells.

In accordance with the algorithm, the following steps are performed toperform the segmentation:

-   1. A remap of the raw input image is first calculated. FIG. 9 b    shows a completed remap of FIG. 9 a . The Merlot remap is computed    as follows:    -   a. A remap image is created of the same size as the raw image        and all its values are set to zero;    -   b. an elliptical, rectangular or other polygon-shaped mask is        formed. A 10×10 elliptical mask is used for the remap computed        in FIG. 9 b ;    -   c. the mask is centered over each pixel in the raw image;    -   d. a gray scale histogram is created from the pixels under the        mask;    -   e. a count of how many bins in the histogram hold a value of 1        or greater is accumulated; and    -   f. the calculated count values for all of the pixel locations        replace the zero values at their corresponding pixel positions        in the remap image.-   2. A threshold is calculated using Equation 1 below and the    algorithm remap image is thresholded to produce a binary image. Such    an image is shown in FIG. 9 c .-   3. Optionally finding the cell colony contours in the image, as    shown in FIG. 9 d , by the thresholded image superimposed on the raw    image.

$\begin{matrix}{\text{Threshold =} - 0.22009\mspace{6mu} \times \mspace{6mu}\left\lbrack \text{Mean image gray level} \right\rbrack - 51.7875} & \text{­­­Equation 1:}\end{matrix}$

The slope and offset of Equation 1 were calculated using linearregression for a set of values, where the mean gray scale level of eachsample image was plotted on the vertical axis and an empiricallydetermined good threshold value for each sample image was plotted on thehorizontal axis for a sample set of images that represented thevariation of the population. The linear regression performed to setthese values is shown in FIG. 11 .

The well metrics are accounted for in the algorithm as follows. Assumesome finite-size region R ⊂ Z. For a random variable X taking on afinite amount of values, the max-entropy or Hartley entropy H₀(X)represents the greatest amount of entropy possible for a distributionthat takes on X’s values. It equals the log of the size of X’s support.

A scene S is a map chosen randomly according to some distribution overthose of the form f : R → { 1, ..., N }. Here R represents pixelpositions, S’s range represents possible intensity values, and S’sdomain represents pixel coordinates.

A Shannon entropy metric for scenes can be defined as follows:

$\begin{matrix}{\text{H}\left( \text{S} \right)\mspace{6mu}:\mspace{6mu} = - {\sum{\text{P}\left( {\text{S}\left( \text{r} \right) = \text{i}} \right) \cdot \log\left( {\text{P}\left( {\text{S}\left( \text{r} \right) = \text{i}} \right)} \right),}}\mspace{6mu}\text{i} = 1.\mspace{6mu}\text{r \textasciitilde Uniform}\left( \text{R} \right)} & \text{­­­(2)}\end{matrix}$

In Equation 2, ~ means ‘distributed like,’ and 0log(0) is interpreted as0. H(S) represents the expected amount of information conveyed by arandomly selected pixel in scene S. This can be seen as a heuristic forthe amount of structure in a locale. Empirical estimation of H(S) froman observed image is challenging for various reasons. Among them:

-   If intensity of a pixel in S is distributed with non-eligible weight    over a great many possible intensities, then the sum is very    sensitive to small errors in estimation of the distribution;-   Making the region R bigger to improve distribution estimation    reduces the metric’s localization and increases computational    expense; and-   Binning the intensities (reducing N) to reduce possible variation in    distributions makes the sum less sensitive to estimation error, but    also makes the metric less sensitive to the scene’s structure.

Instead of estimating Shannon entropy, we estimate a closely relatedquantity. We choose a threshold t > 0 and form a statistic M(S; t):

$\begin{matrix}{\text{N}\mspace{6mu}\text{1}\mspace{6mu}\text{M}\left( \text{S; t} \right)\mspace{6mu}: = {\sum\left| \left\{ {\mspace{6mu}\text{r:S}\left( \text{r} \right)\text{=i}\mspace{6mu}} \right\} \right|} \geq \text{t i=1}\text{.}} & \text{­­­(3)}\end{matrix}$

where |.| is set size and l_(p) equals 1 if proposition P is true and 0otherwise. Now log M(S; t) can be interpreted as an estimator for aparticular max-entropy, as defined above, for a variable closely relatedto S(r) from Equation 2. In particular it is a biased-low estimator forthe max-entropy of S(r) after conditioning away improbable intensities,threshold set by parameter t. Very roughly, Shannon entropy represents‘how complex is a random pixel in S′?’ while log M(S;t) estimates ‘howmuch complexity is possible for a typical pixel in S?’. The describedremap equals M(S; 1) and we can calculate a good threshold for M(S; 1)that is closely linearly correlated with stage confluence.

This algorithm is used to perform the pre-processing to create thecolony segmentation that underlies the iPSC colony tracking that ispreferably performed in phase contrast images. For cells that do nottend to cluster and/or are bigger another algorithm is used, as shown inFIG. 12 wherein we perform the segmentation (cell counting andconfluence calculation) using the bright field image stacks (notindividual images) with a technique for picking the best focus image ina bright field stack.

In accordance with the algorithm, the following steps are performed:

1. Given a stack of images, we calculate a new image that holds thevariance (or range or standard deviation) of each pixel position for thewhole stack. For example, if we have a stack of nine images, we wouldtake the pixel gray scale values of the pixels at position (0, 0) forimages 0-8, calculate their variance and store the result in position(0,0) for what we call the “variance image”. We then do that for pixel(0, 1), (0, 2), ..., (m, n).

2. The pixels with the highest variance are the ones that have differentvalues across the whole stack. We threshold the variance image, performsome segmentation, and that creates a mask of the pixels that are darkat the bottom of the stack, transparent in the middle, and bright at thetop of the stack. These cells represent transparent objects in theimages (cells). We call this the “cell mask.” The cell mask is shown asthe contours in the FIG. 12 .

3. We next create an “average image” of all the image in the stack. Eachpixel position of the average image holds the average of all the pixelsfor its corresponding position in the image stack.

4. Then, we calculate the median pixel color of all the pixels that areNOT on the mask for all and if a pixel in the average image is darkerthan a “darkness threshold” value or brighter than a “brightnessthreshold” value, it is changed to the median value. The average image,when it has been modified in this way is called the “syntheticbackground image”

5. We then calculate the grayscale histogram of the synthetic backgroundimage (shown as the curve 121 on the graph at the bottom left of FIG. 12).

6. We then calculate the grayscale histogram of the pixels under thecell mask (shown as the histogram 122 on the graph at the bottom left ofFIG. 12 ).

When the shape of the histogram 122 is closest to the shape of the curve121, that is the point when the cells have disappeared (they aretransparent, so the best focus point is when they disappear). This iswhat we call “best focus”. The matching of the two histograms issignified by the height of line 123. When the best match occurs, theheight of line 123 is at a maximum. The cells below the best focus aredark and the cells above the best focus are bright.

We can then use this knowledge to create hybrid images well suited forcounting cells, evaluating morphology, etc. The graph on the bottomright of FIG. 12 represents the amount of difference between the cellshistogram and the synthetic background histogram. The minimum of thatcurve at 124 is the position of the best focus image.

One or more imaging systems may be interconnected by one or morenetworks in any suitable form, including as a local area network (LAN)or a wide area network (WAN) such as an enterprise network or theInternet. Such networks may be based on any suitable technology and mayoperate according to any suitable protocol and may include wirelessnetworks, wired networks, or fiber optic networks.

In another embodiment, the cell culture images for a particular cultureare associated with other files related to the cell culture. Forexample, many cell incubators and have bar codes adhered thereto toprovide a unique identification alphanumeric for the incubator.Similarly, media containers such as reagent bottles include bar codes toidentify the substance and preferably the lot number. The files of imagedata, preferably stored as raw image data, but which can also be in acompressed jpeg format, can be stored in a database in memory along withthe media identification, the unique incubator identification, a useridentification, pictures of the media or other supplies used in theculturing, notes taken during culturing in the form of text, jpeg or pdffile formats.

In one embodiment, an app runs on a smartphone such as an IOS phone suchas the iPhone 11 or an Android based phone such as the Samsung GalaxyS10 and is able to communicate with the imager by way of Bluetooth,Wi-Fi or other wireless protocols. The smartphone links to the imagerand the bar code reader on the smartphone can read the bar code labelson the incubator, the media containers, the user id badge and other barcodes. The data from the bar codes is then stored in the database withthe cell culture image files. In addition, the camera on the smartphonecan be used to take pictures of the cell culture equipment and media andany events relative to the culturing to store with the cell cultureimage files. Notes can be taken on the smartphone and transferred to theimager either in text form or by way of scanning written notes into jpegor pdf file formats.

The various methods or processes outlined herein may be coded assoftware that is executable on one or more processors that employ anyone of a variety of operating systems or platforms. Such software may bewritten using any of a number of suitable programming languages and/orprogramming or scripting tools and may be compiled as executable machinelanguage code or intermediate code that is executed on a framework orvirtual machine.

One or more algorithms for controlling methods or processes providedherein may be embodied as a readable storage medium (or multiplereadable media) (e.g., a non-volatile computer memory, one or morefloppy discs, compact discs (CD), optical discs, digital versatile disks(DVD), magnetic tapes, flash memories, circuit configurations in FieldProgrammable Gate Arrays or other semiconductor devices, or othertangible storage medium) encoded with one or more programs that, whenexecuted on one or more computing units or other processors, performmethods that implement the various methods or processes describedherein.

In various embodiments, a computer readable storage medium may retaininformation for a sufficient time to provide computer-executableinstructions in a non-transitory form. Such a computer readable storagemedium or media can be transportable, such that the program or programsstored thereon can be loaded onto one or more different computing unitsor other processors to implement various aspects of the methods orprocesses described herein. As used herein, the term “computer-readablestorage medium” encompasses only a computer-readable medium that can beconsidered to be a manufacture (e.g., article of manufacture) or amachine. Alternately or additionally, methods or processes describedherein may be embodied as a computer readable medium other than acomputer-readable storage medium, such as a propagating signal.

The terms “program” or “software” are used herein in a generic sense torefer to any type of code or set of executable instructions that can beemployed to program a computing unit or other processor to implementvarious aspects of the methods or processes described herein.Additionally, it should be appreciated that according to one aspect ofthis embodiment, one or more programs that when executed perform amethod or process described herein need not reside on a single computingunit or processor but may be distributed in a modular fashion amongst anumber of different computing units or processors to implement variousprocedures or operations.

Executable instructions may be in many forms, such as program modules,executed by one or more computing units or other devices. Generally,program modules include routines, programs, objects, components, datastructures, etc., that perform particular tasks or implement particularabstract data types. Typically, the functionality of the program modulesmay be organized as desired in various embodiments.

While several embodiments of the present invention have been describedand illustrated herein, those of ordinary skill in the art will readilyenvision a variety of other means and/or structures for performing thefunctions and/or obtaining the results and/or one or more of theadvantages described herein, and each of such variations and/ormodifications is deemed to be within the scope of the present invention.More generally, those skilled in the art will readily appreciate thatall parameters, dimensions, materials, and configurations describedherein are meant to be exemplary and that the actual parameters,dimensions, materials, and/or configurations will depend upon thespecific application or applications for which the teachings of thepresent invention is/are used. Those skilled in the art will recognizeor be able to ascertain using no more than routine experimentation, manyequivalents to the specific embodiments of the invention describedherein. It is, therefore, to be understood that the foregoingembodiments are presented by way of example only and that, within thescope of the appended claims and equivalents thereto, the invention maybe practiced otherwise than as specifically described and claimed. Thepresent invention is directed to each individual feature, system,article, material, and/or method described herein. In addition, anycombination of two or more such features, systems, articles, materials,and/or methods, if such features, systems, articles, materials, and/ormethods are not mutually inconsistent, is included within the scope ofthe present invention.

The indefinite articles “a” and “an,” as used herein in thespecification and in the claims, unless clearly indicated to thecontrary, should be understood to mean “at least one.”

The phrase “and/or,” as used herein in the specification and in theclaims, should be understood to mean “either or both” of the elements soconjoined, e.g., elements that are conjunctively present in some casesand disjunctively present in other cases. Other elements may optionallybe present other than the elements specifically identified by the“and/or” clause, whether related or unrelated to those elementsspecifically identified unless clearly indicated to the contrary. Thus,as a non-limiting example, a reference to “A and/or B,” when used inconjunction with open-ended language such as “comprising” can refer, inone embodiment, to A without B (optionally including elements other thanB); in another embodiment, to B without A (optionally including elementsother than A); in yet another embodiment, to both A and B (optionallyincluding other elements); etc.

As used herein in the specification and in the claims, “or” should beunderstood to have the same meaning as “and/or” as defined above. Forexample, when separating items in a list, “or” or “and/or” shall beinterpreted as being inclusive, e.g., the inclusion of at least one, butalso including more than one, of a number or list of elements, and,optionally, additional unlisted items. Only terms clearly indicated tothe contrary, such as “only one of” or “exactly one of,” or, when usedin the claims, “consisting of,” will refer to the inclusion of exactlyone element of a number or list of elements. In general, the term “or”as used herein shall only be interpreted as indicating exclusivealternatives (e.g. “one or the other but not both”) when preceded byterms of exclusivity, such as “either,” “one of,” “only one of,” or“exactly one of.” “Consisting essentially of,” when used in the claims,shall have its ordinary meaning as used in the field of patent law.

As used herein in the specification and in the claims, the phrase “atleast one,” in reference to a list of one or more elements, should beunderstood to mean at least one element selected from any one or more ofthe elements in the list of elements, but not necessarily including atleast one of each and every element specifically listed within the listof elements and not excluding any combinations of elements in the listof elements. This definition also allows that elements may optionally bepresent other than the elements specifically identified within the listof elements to which the phrase “at least one” refers, whether relatedor unrelated to those elements specifically identified. Thus, as anon-limiting example, “at least one of A and B” (or, equivalently, “atleast one of A or B,” or, equivalently “at least one of A and/or B”) canrefer, in one embodiment, to at least one, optionally including morethan one, A, with no B present (and optionally including elements otherthan B); in another embodiment, to at least one, optionally includingmore than one, B, with no A present (and optionally including elementsother than A); in yet another embodiment, to at least one, optionallyincluding more than one, A, and at least one, optionally including morethan one, B (and optionally including other elements); etc.

In the claims, as well as in the specification above, all transitionalphrases such as “comprising,” “including,” “carrying,” “having,”“containing,” “involving,” “holding,” and the like are to be understoodto be open-ended, e.g., to mean including but not limited to.

Only the transitional phrases “consisting of” and “consistingessentially of” shall be closed or semi-closed transitional phrases,respectively, as set forth in the United States Patent Office Manual ofPatent Examining Procedures, Section 2111.03.

Use of ordinal terms such as “first,” “second,” “third,” etc., in theclaims to modify a claim element does not by itself connote anypriority, precedence, or order of one claim element over another or thetemporal order in which acts of a method are performed, but are usedmerely as labels to distinguish one claim element having a certain namefrom another element having a same name (but for use of the ordinalterm) to distinguish the claim elements.

It should also be understood that, unless clearly indicated to thecontrary, in any methods claimed herein that include more than one stepor act, the order of the steps or acts of the method is not necessarilylimited to the order in which the steps or acts of the method arerecited.

What is claimed is:
 1. A method for processing an image of cell coloniesto segment the cell colonies, comprising the steps of: providing a grayscale image of cell colonies comprising a plurality of pixels; remappingthe image; centering a mask over each pixel in the remapped image forcreating a gray scale histogram; calculating a number of bins in eachhistogram, wherein the total number of bins is less than required by thegray scale range; assigning the calculated number of bins for a pixel asa value for that pixel in the remapped image; thresholding the remappedimage pixel values to produce a binary image wherein the cell coloniesare segmented.
 2. The method of claim 1, wherein the step of calculatingincludes using only the number of bins having at least a minimum valueto filter out noise.
 3. The method according to claim 2, wherein theminimum value is
 1. 4. The method according to claim 1, wherein the stepof thresholding comprises using dynamic thresholding.
 5. The methodaccording to claim 4, wherein the dynamic thresholding is a function ofthe mean gray scale level of the image.
 6. The method of claim 1,wherein the total number of bins is 32 for a gray scale image having arange of 0 to
 255. 7. The method of claim 1, wherein the mask iselliptical.
 8. The method of claim 1, wherein the mask is polygonal. 9.The method according to claim 1, further comprising calculatingconfluence using the binary image.
 10. The method according to claim 1,further comprising determining texture from the binary image to identifyundifferentiated and differentiated cells.
 11. The method according toclaim 1, further comprising calculating a cell count from the binaryimage.
 12. The method according to claim 1, further comprising applyingthe binary image to a backlight display to align light from the displaywith the imaged cell colonies.
 13. A method for processing an image ofcell colonies to segment the cell colonies, comprising the steps of:providing a stack of images of cells; calculating a variance image thatdisplays the variance of each pixel position for the whole stack,wherein pixels with the highest variance are the ones having differentvalues across the whole stack; thresholding the variance image to createa cell mask of the pixels that are dark at the bottom of the stack,transparent in the middle, and bright at the top of the stack; formingan average image of all the images in the stack, wherein each pixelposition of the average image holds the average of all the pixels forits corresponding position in the image stack; forming a syntheticbackground image by calculating the median pixel color of all the pixelsthat are not on the mask and if a pixel in the average image is darkerthan a darkness threshold value or brighter than a brightness thresholdvalue, it is changed to a median value; calculating a syntheticbackground grayscale histogram of the synthetic background image;calculating a pixel grayscale histogram of the pixels under the cellmask; and determining a best focus when the curve of the pixel grayscaleyellow histogram is closest to the curve of the synthetic backgroundgrayscale histogram.